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ABSTRACT ■ * * 

Title XX funds are. used at the state level to 
develop, expand and improve the training of social service workers- 
Evaluation of .programs receiving this f unAiaagjnust provide , 
information on the effectiveness of the program in meeting 
specifically stated objectives, a complete, description of the program 
for the purpose of replication, and recommendations fcr program • 
improvement. Two core evaluation procedures are- the development* and 
use of behavioral Objectives or qoal statements and the development 
of a detailed description or chronicle of all procedures used or 
events occurring during the course of the graat period. Additional 
evaluation procedures used either to supplement or validate the core 
procedures are criterion referenced pre- and post- testing , self, 
reporting of ft kills, use of attitude change scales, behavioral - 
observation aid participants' subjsctdve ratings. 'Recommendations for 
f the- format in summarizing the collected information are given. y . 
'Questionnaires are appended. (BH> 
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there are 500 Individual Title XX eligible social service agencies. 

These agencies include day care centers, prison's, senior citizens centers, 
regional welfare and health centers, probation departments, Halfway houses, 
financial counseling services., alcohol rehabilitation and ^ job placement 
agencies, and visiting nurse services'rn addition to jiftrious*' other agencies. 
The ovei^6,q00 social service providers employed by tfiese agencies* come from', 
a wide range of educational backgrounds, Approximately 15 percent of the 
employees have not completed* the equivalent of a high school education, 
53 percent have a high' school diploma, 19 percent have completed an under- 
'graduate college degree program, and about 13 percent have a graduate 
degree ( f iguresr^ased upon* 19 77 estimates). 

Bearing in mind the. scope of training' funded through Title XX monies 
and the heterogeneous nature? of the population served, it is clear that 

* * 

various evaluation procedures need to be implemented to chronicle > the 
development and effectiveness of training programs. Rather thaji relying on 
-a single formula for evaluation, many types of evaluation techniques are used 
in concert. The" 'dif ferent evaluative procedures .pro jf/e meaningful data for 
social. .service .agencies^, program developers, program participants , and for 
the funding office itself. In constructing the evaluations, the needs of 
each data recipient are critical in determining the types of procedures used. 
Social service agencies need data describing what skills each of their 
employees ha've and wilThave a* the result -of training. Program participants 
need information ngparding what training is or will be available and hoy^ 
such training canTit with their job related needs. The program de^lopers 
and funding office need to know how effective a program*has been, -'how to 
improve effect iveness* and how to replicate the desirable outcomes. The ' 
procedures described- in the rest of this'document are a sample of the types 
of methods used tcj meet the expressed evaluation needs of Title XX grants. 

Core Procedures . 

Many of the specific evaluation procedures implemented on Title XX 
grants are "individualized" to fit ne^ds and goals 9f the different grants, 
b.ut there are two procedures which are constant throughout all grants funded. 
Jhese two ^core procedures are the development and use of behavioral objectivef 



• ) • • • 

v or goal statements ana .the development of a detailed description or chronicle 

of all procedures used or events occurring during the course of then gr^rlt 

period* These two core p^cedyres provide the backbone for all subsequent 

vevaluation procedures. They have^een found tfo be. invaluable when attempting 

to provide information for agencies anii participants as well as for 

» * 

dissemination and replication. * * ^ 1 

Statement of Goals ' 

m ' 

While not necessarily thought of as a traditional method jof evaluation, 
behavioral objectives, or specific goal statements, have* been found to play a 
critical part in Title XX grant evaluations. The importance of specific 
objectives are best summarized by Robert Mager (19 75)" . if you're not sure 
where you're goinjg, you're liable to end up someplace else (preface)'. " In 
the case Osf Title XX grants, if principal invest igators (academics in most 
cases), agency personnel (both* administrative and direct service providers)* 
and the evaluator(s) do not know the specific intent or goals of a given 
grant, eacl? party may end up in. a different place and with a different degree 
of satisfaction as a result of having original impressions of goals met or 
unmet at the conclusion of a grant. ItS is not uncommon for an academic 
acting as aprincipal investigator on a grant for a first time to' assume that 
hi& audience will* easily understand content specific jargon or broad, sweeping 
statements describing potren tial ^outcomes . It is similarly likely that^ 
administrative personnel and service providers in agencies enter training 
programs provided as a result of a grant with preconceived expectations frhich 
may be Independent of* actual training content. Hence.it is imperative to 
generate specific descriptions of what outcomes will result from> the implementa 
tlon of each grant. * 

The specifically seated objectives also provide a base from which an v 
evaluator should build all subsequent evaluation procedures. The statement of 
objectives should provide a description of the behaviors, attitude changes, - 
service changes or permanent products which should be evident at the con- 
clusion of the grant if the grant staff has been effective in meeting stated * 
goais. - Ih other words, statements of the ob jeffctives pi a given grant should 
provide -the basic framework for the evaluaticjn. They*should identify all 
components of * the grant vhich wa rrant Inspec Wbn to determine, effectiveness 



as well as suggesting a means t for revaluation of the degree of effectiveness* 
"For example, a specific objective of a grant funde.d for the development of 
training^ Jfiims may sta\jCL. that eight color videotapes will be produced. 
In that> case, it would be alear that m the first step in the evaluation w^uld^b< 

determine if all eight films were produced by the conclusion of the grant 
period! If f&ver than eight were ^produced, or if some were produced in 
black and white rather than color, then the grant has been less than 100%/' 
effective in meeting stated goajs. It is then clear that certain recommenda- 
tions (i.je., a revision in filming schedule)' are necessary before replication 
may be 'considered.- • 

This very straightforward example , may' seem even to obvious* to bother 
to state, 'but that has not been found to be the case. It is possible f° r ^\ 
grant proposals to Include vaguely stated goals as "appropriate training 
tapes will be developed. "\ .In such an Instance it is unclear how many and 
what kind of tapes will Jbje developed, how long the tapes will be, etc. The 
specific .objectives help eliminate any inappropriate expectations. 

A second example of the usefulness of objective goal statements is 

illustrated by a proposal to train agency personnel In the area of consumer 

rights and responsibilities. One of the original goals of <k proposed 

grant had been^ stated M t,o provide training for service providers in the area 

of Food Stamps and Food Issues." As stated, that; goal was subject to a' wide 

range (jjf interpretation^ . As finally described in a more specific goal state 

ment^ the- intent of the training* was Identified as "to prepare the service 

provider to explain to clients, (1) new regulations regarding the Revised 

Food Stamp Eligibility Formula, <2) eligibility requirements fo<r Food Stamps, 

(3) procedures involved in securing Food Stamps,,.." The specific restate- 
• * * 

ment of the training goal helped eliminate any agency misunderstandings of 
what skills stajr£ workers would develop as a result of training. Logical 
evaluation procedures (for example, observe participants explain regulations, 
eligibility requirements^ procedures, etc. to clients) were also implied by 
the goal statements., ^ 

Aside from establishing specific and accurate expectations for the 
results of funded .proposals and providing a framework for subnequent 
evaluation procedures, objective goal .statements facilitate publicity ef forth 
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in recruitment of participants and ease dissemination efforts. If well 
stated, instructional objectives/may be included fn *brach tires and flyers, 
used to publicize classes and workshops • When used in this way th6 
objectives are effective and efficient tools for communicating expected 
instructional outcomes to both the direct service providers and administra- 
tive personnel in Title *XX agencies. Upt>n completion oX^any grant these 
same objectives may 6e* ased to communicate tq Title XX agencies* in other 
states the types of goals which have been achieved througlr funding efforts. 
In that way, agencies in other states may capitalize upon the national 



nature *>f Title XX funding. An agency in California, seeing the stal 
objectives of a grant funded in Connfecti : cut may decide that those objectives 
fit existing training needs. At that point, time and effort may be saved 
and effectiveness may be improved by replicating * the grant^as it \^as 
funded iti Connecticut and implementing any recommendations from the 
original grant. Hence, dissemena tion ^and replication of Title XX grants 
are enhanced by the use of objective goal statements*/ ♦ 

Detailed Description of Procedures and Events t * 

The^ second" core procedure, implemented on all grants is the requirement 

oj a detailed chronitle of the development of the grant and any instruction 

■ i 

funded through the grant. The parts or requirements of these chronicles 
are adapted to the nature of * the grant receiving funding. In the case of a 
. rfeeds assessment and curriculum developntent grant, a description of the 
development of any assessment questionnaires used, a copy of questionnaires 
used, a written description of the procedures used in interviews, dates of 
all interviews or mailings, a list of agencies and individuals contacted, a 

list t of agencies and individuals responding, an estimate of the response 

" : ' ~ * 

rate (as *a percentage of the targeted- population for the assessment), a 

( ' \ + ' 

description (5f the stratification of the sample responding to an assessment 

and a summary of the results of an Assessment ape required; The descriptions* 

pertaining specifically to the curriculum development portion of a grant must 

include an Identification of any existing curriculum similar to the one 

developed and where the existing ono(s) may be found, any curriculum 

pb^erved during the course of developing the new one, any references used 

during the course of curriculum development, the materials (tapes, journals, * 
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audiovisual equipment, etc.) which would be required to implement the. new' 
curriculum, the names -of any persons known to be skilled in that particular 
content area and prepared to implement the curriculum, and a description of 
the audience for whom the curriculum was developed. When classroom or 
field instruction are~the -goals pf the grant, descriptions should include 
the names and affiliations of all instructors and guest speakers, the names, 
agency affiliation and general demographic information- pertaining to' all 
participants, 'the time and loc^ion at which the class pr workshop was. 
conducted, and an. annotated course outline as actually implemented {including 
-• all handoAs and overheads used, a lJLst of reference? materials used, a 
bibliography' of related readings and ref erence ^materials, a description of 
all activities used, etc.). In all cases, be it assessment and curriculum 
development or any kind of instruction, the names, agency affiliation and 
job description of any personnel involved in the actual grant itse,lf must 



be Included. 



.These descriptions serve two-purposes. First they provide all 
iQXprmation necessary for a' repl ication of the grant.- The^propriate 
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ence has been defined and the technical workings of the grant are *"*' 

/identified. From these descriptions interested Title XX funding agencies^ in 

• « * * 

other states or other Title XX^ coordinating- of f ices should be able to " ' 

implement similar programs with predictable results ^ In addition, the 

descriptions" provide data for accountability purposes, 'identifying Itho was 

responsible for what* and what actually occurred as a result ,of . fun iing. 

t • The second use of the descriptions 'Is to provide names and agencies 

served for fhe potential user of the programs developed. The potential 

user, either another Title XX coordin^ng office, an agfency, or an individual 

service provider, shoul d have sufficient information fro/ these descriptions 

to evaluate program procedures for appropriateness in meeting existing 

,needs. The names .of both, grant staff and'program participants should also 

give the potential 7 user a head start if he or she wishes to get additional ' 

impressions, from those directly involved in the - grant. I£ one has any 

questions remaining following reading of an evaluation report, these detailed 

descriptions should provide direction as to* whom should be contacted to 

secure answers to questions. . N ^ 
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A caution to the reader is necessary at this point. It has been the' 
experience of this evaXuator to find it difficult tp secure detailed . • 

instructional outlines. This has been found .to be the case in particular 
when well known content experts have been invited to address a group of 
Service providers during a workshop or workshop series. Unless contingencies 
are arranged in advance, workshofNleaders may be reluctant to submit revised 
outlines of material^actually covered during instruction. It appears "that 
one of* the failings of human A|ture is to assume that once payment for 
instruction is -received, no further responsib il it ie^. exist . In light of t 
past experiences, the Title XX* office at the University of Connecticut is 
•experimenting with modified contingency "contracts. In some cases, -"particu- 

• a 

\ 

larly when national experts *or invited speakers are included* in classes or 
workshops, contracts are written with receipt of paynjtent contingent upon- 
the principal investigator or coordinating office reviving certain . 
documents. For example, an expert may receive 25 percent of his total 
honorarium following ^submission of specific behavioral objectives for a given 
instructional unit, another 25 percent following provision of a proposed 
instructional outline, 25 percent following actualv^infct ruction and the 
final 25 percent upon receipt of A revised, annotated instructional outline* 
Again, this procedure is currently being sxplored^ so its effectiveness and 

limitations are not yet clear. ' , 

• > * 

Additional Pro cedu res * * , 

In almost all cases additional evaluation procedures, are used to either 
supplement* or validate the 'core, procedures/ The most appropriate procedures 
to include on each separate grant are chosen from a group or pool of' methods 
to be discussed in this section. Selection ,of procedures is based upon the 
specific behavioral objectives stated foi 1 each project. » 

< 

Criterion-Referenced Tests ' 

* 

Whenever possible, a pre- and post^ instructional administration q^-Zan^ 
objective, content based test is used -to ass€^& change in knowledge or skill 
evidenced by participants* Crite r Lon refere^etf tests are used for this, 
purpose. As suggested by'Popham (1978) cri tericn^re ferenced tests are 
probably tfte best tools available to secure specific information describing 



what a person can or cannot do. The use of criterion-referenced testfe also 

" makes it possible to closely match test content with actual instruction in 

classes , seminars or workshops* 
* 

The use of cri terion-referenced pre- and post-testing directly serves 
two of the three purposes underlying the evaluation' of Title XX funded 
projects. First," by 'comparing pre- to post- tests •♦scores," it is easy to 
discern a 'change in knowledge or skill level. This information pertains 
directly to determining the effectiveness of instruct ionally oriented grants 

meeting objectives.' A plated benefit ot such testing is that if the 
oriterion-*referenced tests are appropriately developed to give 'good content 
sampling of the entire instructional domain, post-tests may serve to provide 
evidence of skills participant^ have following instruction. ^ 

The second use of scores from pre- and po^t-tegting is for instructional 
improvement; If we refer to Bloom's (1976) basiq model of instruction/ 

it is cleaTr tlhat training is most effective when teachers have a good sense 

' ' - "1 * f 

of the entering abilities of course or workshop participants. Knowledge of . 

the characteristics of the audiince allows for instruction to be adjusted 

to the most appropriate level for participant understanding. Unfortunately, 

we cannot all* sense audience needs a priori. The use of a pre-test allows 

for a systematic collection of data describing the entering abilities of 

participants. If these data are collected in advance, changes in planned 

instructional techniques or 'content may be ma<fe to best fit audience 'needs* 

But data are not always collected in sufficient time to allow for 

revisions in instruction -before implementations In those instances^. it is 

the post^test da*ta which provide the information for improving the 

effectiveness of instruction. Through the use of item analysis, content which 

was* made understandable to only a portion *of the audience or content which 

was misunderstood by vail the participants is identified. Based upon the 

combined results ^of prQ-and post— test administrations of criterion referenced 

tests, specific revisions in instructional content for an explicitly defined 

audience can and , shoal d be derived. These recommendations are provided for 

agencies considering requesting a replication of the program, for principal 

investigators responsible f<y£ a replication, and for other coordinating offices 

reviewing training efforts. m ^ \ 
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Sel£ Report of Skills- 

A procedure often used in conjunction with/criterion- referenced tests 

/ 

is participant self report of skill levels ♦ When cdupled.with criterion- 
referenced tests, self report may be used to cross validate post-test scares 
as well as tQ generalize estimates of knowledge g&ined in broad content 
domains through the use of a limited number of questions. , The use of self 
report without a criterion-ref erenced test, while less preferable than an 
♦objective evaluation, has been turned to in some specific instances. In 
some workshops the content presented does not fit *easily into an objective 
testing model. For example, when presenting training to human service 
providers in areas sylcH as how, to influence "staff burnout' 1 in mental 
institutions, or how to deal with clients problems in the emotional areas 
of death and dying, content based objective tests se^n' awkward and potentially 
offensive. In those instances the^elf report measures, while clearly not 
as 4 systematic, objective or unbiased and valid as criterion-referenced tests, 
do help proAtide information about participants' sense of knowledge gained. 

v A quite different instance when self report of knowledge or skills 
gained appears appropriate is where a needs assessment has been conducted 
prior to training. c Potential participants of Title XX training programs 
are typically asked to report on their "present" and "desired" skill level 
in a given domain as part of a needs assessment to determine training 

priorities (see Appendix A for an illustration of such an instrument ). 

*> 

In those situations, a logical conclusion to training efforts is td ask the 

•0 4 

audience to again rate their "present" and "desired" skill levels in specific 

'content domains* If post^instruct ional self ratings of skills are of the 

• v 

same level as the pre-inst ruct ional needs assessment, then training has been 
less than effective.* On'the 6ther hand, if "present^and "desired", levels 
of skills are approaching the same ratings following participation, the- *\ 
training' has proBably been appropriate. , 

Attitude Change Scales 

While a change in skills or awo^crease in knowledge of content by 
service providers is the primary goal of most Title XX training, other 
subsidiary and often less tangible goals are also pursued. Gluen the » 

- •' 11 
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nature of the jdb performed by the Title *XX employe, ^ to provide human 
services to clients, a#change in attitude is ofeteru ^||p:ical to insuring 
optimal grant effectiveness. Illustrations pertaining \to client concern 
for death and dying W<J how to ^ealrwith "staff burnout" or decrease in 
motivation on the part ; of the direct care provider have already been 
mentioned. Another area when attitude change ma/ have a critical impact on 
th$ effectiveness of training is marital and family therapy. In order 
to \>e an effective family or marital counselor, it is imperative to 
understand multiple possible-solutions to family 4nd/or^ marital problems 

In o?der to understand those possible ' solutions > it seems important'to 

, * j 

- approach the alffernat ivls with an open and unbiased attitude. In other 
words, Title XX providers anust often put asi&e preconceived attitudes in 
order to be effective learners and/or human service providers. Thus, 
attitude change is often a goal of Title XX funded- projects. 

The area of measuring attitudes and attitude change is the most 

A difficult domain within the TifcJLe XX evaluation system. Attitudes, by 
nature of being a construct, are difficult t tp define and measure' When 
possible, previously developed attitude scales* are used ; Two references 
wh ich* have been of great help in locating scales amenable,. Jjp measuring * 
anticipated changes in participants.' attitudes are Measures of Social 
Psychological Attitudes (Robinson and Shaver. 1978) and Measures of 




Shave 



Occupational Attit udes and OccupationajWharac t^ristics X Robinson, 
Athanasiou, and Headxrl976) . These two Volumes present descriptions of «/ 
the intent, development , validity and reliability evidence and sampled of * 
-items from a large number of established instruments used to me^rfiire 
constructs- These books have often been found to provide ^erfough info i^mat'ion' 
to allow for the selection of a scarle or scaJ es to measure intended outcomes. 
When it is clear that no appropriate scale exij^s^to serve the purposes of 
a given gt;ant evaluation, the Robinson^jerjfcil . references have been useful 
in providing item and test formats' which have been appropriate for measuring 
related constructs, ther€b^helping to establish a basis or format for a 
scale to'be 'developed within- the course of a particular grant. 

Development of Criterion^Referenced Tests and Attitude- Scales 

When .att it ude ".change scales or cri terion^referenced tests ate developed 
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for use on a grant,, a semi-st ructured insttument development process is 
empl6ye4. Fijrst, specific hypotheses pr* expectations *re . stattd .describing 
what change in attitude*?, skills or knpwledge' of' content is anticipated. 
Once a domain is specified by the hypotheses or expectaricyf statements, items 



which reflect a representative sampling of the domain rfrjFdeye loped. The 
i^teips are. then • reViewed by at least two experts,, one content expert to 
determine *the-face validity of the item, the second reviewer to ditermine 
if ■ the* item format inappropriate (if all grammar is correct* if there«wis % * 

^7^?byiou^^ of a testwlseness cuey within the item, etc,.). _ 

Following the . revision of 'any items or scales which Show evidence of proSlems 
during the first review, the items are combined into a test or scalfe and, if 
time permits, piloted with a group representative of the anticipated audience. 
When the instrument being developed is a criterion-referenced test, 
responses are % submitted to an item analysis. Based upon the difficulty an< 
discrimination indices resulting from the item analysis, individual items 
are again revised when necessary and put into final form, ;When the scale 
in- question is being developed to measure attitudes, a factor analyses of 
the pilot data* is conducted. Deletion of 'items, incorporation of new or 
additional items is then based upon the factors which emerge from the 
an&lysis . * \ i \ 

The major problems encountered in scale development for Title XX 
evaluations pertain to item writing and piloting of scales. Development of 
good items is contingent upon at least two factors - knowledge of the 
boundaries of the domain to be sampled and an understanding of item, writing 
procedures. As alluded to previpysly , it is sometimes difficult to secure 
specific outlines of coritent and goal\stat6merits from workshop presenters or 
classroom instructors well in advance of actual training. This difficulty 
has led to experimentation with established contingencies ('for example, 
25 percent. of total payment is released following receipt of either an 

' * r 

in-depth outline or some sample test items for a criterion-referenced' 
instrument) within contract agrfeemente with content experts. The other 
problem, that of securing well, written items, or, persons with good item writing 
skills, remains unresolved. It appears that item* writing is as much an 
art as a science and that expertise comes with practice. These skills are 
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generally absent from the large repertoire of skilfs of trainers and field 
instructors on ^grants, as well may be expected. The content experts * * 

• • *■ V 

employed by Title XX- projects are primarily concerned with* imparting skills 

and knowledge to direct care providers. These experts have little time to 

concern themselves with how, to'write a multiple choice item, what descriptors 

to include in a semantic dif ferentialj or how to limit the influence of >, 

social desirability in responses. Unfortunately, this problem has meant x 

that in most cases an item writ ing^exp^rt i^s. either forced to work closely 

and for many hours with the content expert or to become a content expert 

himself.' Again, thi,s problem remains unsatisfactorily resolved at this 

t 

point. . 

The other major problem, piloting newly developed instruments, is more 

time consuming and mechanically frustrating than procedurally difficult. 

The time of service providers is often at a premium ~ they are loaded and. * 

overloaded with responsibilities to clients. It Is. difficult and takes 

many phone calls and hours to s'ecure' a representative audience in advance 

of actual training. Response rate to pilot tests disseminated through the 
* 

mail qt directly to agencies is dishearten ingly low. Because of problems . , 

evident in, trying to secure a truely representative sample, an alternative 

chosen in many cases is to pilot items on captive audiences which are 

readily available (for example, intact college classes, workshop or training 

groups of. Title XX providers, etc.) This alternative is unsatisfactory and 

Efforts are still being made to find other more representative audiences to 

serve as samples for instrument development. 

The problems described above are' the major contributors to , delays in an 

instrument development process.* Ideally, an item pool is developed well y 

enough in advance to allow for necessary- reviews, revisions, piloting, analysis, 

and final revisions of items. Frequently time runs too short to allow for , 

the full and recommended procedures of instrument development. In reality, 

corners sometimes i?eed to #e cut on Title XX inst rument development ventures . 

♦ 

There is no set pattern by which the corners are cut. Judgemental decisions * 

are made by tl*2 Title XX coordinating office stqff In concert with individual 

grant staff about *riiat procedures msy be eliminated with the least amount of 
* * . 

threat to the validity and integrity of those instruments developed. 
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Bfeh&vioral Observation ^ * 

Another procedure used as, a validity check for instruments employed in 
Title* XX evaluations is behavioral observation, Acjxial on tjje job imple- 

s 

^njLentation of newly developed skills offers a handy criteriop to be used as- 

evidence of the concurrent^ validity oj: Content and attitude tests. The most 

easily implemented forms of behavioral observation have been found to be 1 

^ » * 

diaries or logs and supervisor or cohort reports. m4 

JDiaries cjr logs have "been a conyenient w$y for Title XX employees to 
-keep- -track of typical pr^ 

participation in training programs. The * information iri these logs may be 
used for many evaluative purposes . % It can provide examples for trainers to 
u^e which closely reflect typical problem^ dealt vitfi'"*fey Title XX, service 
providers ^ RecoTds of proposed solutions to' client problems reflect an 
understanding or lack of understanding of material presented in training. 
The incidents, recorded in logs and changes in proposed solutions provide 
evidence of imp! emen t a t ion of skills learned £hro'ugh participation in programs 
funded through Title XX funds, the .final goal of training efforts. Finally* , 
information from logs is, used to make revision^ in training curricula and 
recommendations for improvements in instruction. ; % 

tog. or diary entries are subject to influences of social *desirability 
and Image managementT Supervisor or cohort observations of procedures used 
or behaviors exhibited by grant participants have been found helpful in 

adding external checks of? the val idity of self reports in logs and diaries* ; 

3 . r\ ■ 
The major difficulty in securing observations by others has been the time 

restrictions faced by most Title XX employees. As previously stated*- 

Title XX employees are commonly overloaded with client cases and paperwork 

and do not look f avorably^upoh* additional chores^. Keeping requests for 

supervisor or cohort ratings short, specific and to the point has been found 

• ■ * 

to be most advantageous in collecting such data. For example, it is far 

preferable to ask the cohort of a participant If the participant shared skills 

learned in training, has used more group rather than individual therapy with 

clients, ot simply responds quicker to messages than before participation" 

rather thlah askiYig cohorts "Have you noticed any change** in your fellow worker' 

tore structured behavioral observations have been used on various grants 
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with variable degrees of-Osuccess. When conducting field instruction in 
•behavioral techniques with th,e staff at a residential institution for 
mentally retarded clients, structured behavioral observations of ' " * 
^participants 1 interactions witji clients has been found n a successful eval- , 
u&tion tooll When a similar;, technique was attempted in a day. care setting, 
following instruction, the' procedure, failed. While found unobstrusive 
in the institutional setting, -observers were found to be too disruptive to 

*\ * ' 

be maintained in -the day aare facility*. V . 

Inr other instances, . videotapes o*f classes on individual participants 
during the first and final weeks of f raining have provided behavioral 
evidence of change following instruction. The videotape method has been 
found particularly worthwhile for use in sign language classes taught for 
social service providers working with deaf clients. The tapes illustrate 
change in behaviors of signing following instruction as well as giving % 
participants a look at their own effectiveness in non-lingual communication. 

• A final form of behavoral observation experimented rtith'and found 
effective in- Title XX grant evaluation is thv u&e 9f observatipn through 

7i one-way mirror I Following training in- group facilitation dndf group . 

* * ~ 

therapy techniques, Title XX participants were pUt in t simulated -group . . 

therapy situations. Unobtrusive 6bser;vation tjirough^a one-way mirror of 

the methods employed by Title XX participants in structuring and leading 

the groups provided evidence of skills developed f.oll'owing instruction. The 

observations also helped identify skill areas which Still needed practice, 

thus identifying recommendations for* improvements in training content. 

1 A summary of 'the 'use of behavioral observation as an ^valuation tool 

must stress its unparalleled value in providing a criterion for estimating 

the concurrent and const ruct val idity of any other evaluation techniques 

employed. Paper and. pencil measures, interviews and self % reports ^re 

unquestionably more efficient methods of evaluation than is behavioral 

observation. Efficiency of evaluation ,|however, must be carefully weighed 

against the validity of observation. Perhaps the problem is best summarized 

j>y Scheirer (1978) wtren she suggests, "When looking 'for evidence' of 

behavioral change toward program goals, don't believe anyone ! s. impressions 

including your. own. Behavioral 'changes require behavioral "evidence (p. 67). 



. .. . . V 

If the final goal of Title Xtf funding is to improve service l to clients, then 
the behavior»*to be observed following training are services to arid 
interactions ftith clients. 

But behavioral observations are 'probably tfiefmost difficult form of 
evaluation data to collect*. ; Behavibral observation is 1 * costly. Title XX 
agencies are frequently reluctant to open their doors' to outside observers. 

Cohort and supervisor ratings, are r not nearly -so' structured or systematic .' . 

* - •*. » 

as would be hoped for in observation datfc. It is frequently' difficult 

to .target spteci'f ic *behaviors to observe in natural settings. It is . ' 
difficult to know how valid a- generalisation, from a simulated situation to 

on the job setting may be. The list of problems tied 'to behavioral £fc 
•observations gdes on and on. • 

So how may we balance the.&enef its of behavioral observation^ as an '^ 
evaluation tool with its practical limitations evident in reality? This 

question, as- fehii 'orie pertaining to good item writing and securing % 

* ' \, " . . 

representative participant samples remains unresolved. At present, the 

policy of the Title tfX office at the University of Connecticut is to , 

continue* to collect as much o^eyya.tipn data as possible and explore any 

new, unobtrusive, economic and feasible procedures which may be devised. 

* 

Participants' Subjective Ratings ; 

1 » 

' Participant satisfaction questionnaires used V>n all grants funding 
field or classroom instruction., .These instruments are *genSrally very 
straightforward, asking for participants ratings of tfie physical arrange- 
ments of the training, materials used during training* scheduling, etc. 
Such questionnaires have well documented limitations, the greatest <>f which 
is the'tertdancy for participants to r give inflated and unjustly positive 
responses '{Scheirer, 1978), In recognition of the limitations of these 
questionnaires' efforts to determine the concurrent and construct validity 
of responses are, made. Typically, these procedures employ multiple 
regression/ techniques. Oiany scores in knowledge of content (pre-test 
scores/on criterion^fef^xenced tests are subtracted from post- test , scores 
to determine a change scor^) Vpost-tes^t scores, on criterion- referenced «> • 
tests, indices of attitu^ls and, attitude changes in related areas, 
attendance at classes, completion 'of assignments, etc. are commonly used 
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to predict* participant ratings on Satisfaction questionnaires. The extent 
t% which participant 'satisfaction can be accurately predicted and the 
apparent logic of the predictors (for .example, high post- test scores afld 
•,large change scores with regard *to knowledge of Content would intuitively • * 
show a positive correlation with satisfaction scprej>) give support to or 
refute the construct validity of 'satisfaction questionnaire responses. 

Usefulness ratings participants are treated in much the same way 
as are the responses . to satisfaction questionnaires. . Rathdr than asking - 

" r % * } » 

participants fpr their ratings of the comfort, of the seating of ins true tiou^ 
whether they felt*that a class wds scheduled at an appropriate time or if 

cla^s was too large or too long, usefulness questionnaires attempt- to 
determine, how relevant draining was to actual arid anticipated job 
requirements. Whenever possible an investigation of the validity of 
responses to usefulness questionnaires is conducted by comparing the relation- 
ship between usefulness ratings ami othet evidence 0*f grant effectiveness . * 
Tjhe usefulness ratings are primarily for program revisions and as a source • 
of recommendations for improvements. Such rating instruments commonly 
require participants to identify aspects of the training wtilch^were most and 
least beneficial^ endeavors which were most zipplicable to job demands and those 
least relevant to job* demands. These aspects of the ratings help capture 
specific feedback to indivi dualize training-activities to fit specific job 

and ageney demands. . > ? - 

■/ 

A participant rating form^which was developed for] use ini the evaluation^ 
of training programs, particularly those programs conducted in a classroom 

r i / - 

setting, 'is included in Appendix EL. This questionnaire* has gone thrdugh a 

series of revi^ipng. and instrument development procedures. There #re five 

factors underlying the version of the scale presented. The factors 'were 

* 

derived from^a principal components analyses of the data describing, ratings 
from 120 participants in morg than 20 instructional programs. . The five 
factors pertain to. course materials, course content,' course scheduling, the 
.use and uniqueness of the information presented, and the audiovisual aids 
and course -length. The average alpha estimate of the reliability of the 
factors is .69/ Ttie items defining ea'ch ,of the factors are identified in 
Appendix C.__- « * 4 



The primajry use of ratings from the scale has been to provide feedback 
,to instructors' -for the purpose of instructional improvement. Procedures to 
compare/ rating^ of instruction following feedback .of ratings are now being 



exploited. Thej^se of the rating instrument as part of a formative as well 
as a/sumraative ievaluatioh procedure is also being investigated- \ 

Conclusion . / 

/ — •■. . - - < v . / • . 

.1 » The information collected through the use of any/or all of the procedures 
defscribed* above must be summarised upon the completion of each grant. Again, 
the three purposes of the evaluation must be brought to mind. In addition,- 
the potential audience reading the* evaluatio^must /be remembered. Title XX 
agency per so^el, the staff of other Title XX coo/dinating offices and, other 
principal investigators must all be able, to comprehend the evaluation. For 
this reason,] evaluation reports divided into sections are recommended. 
Conceptual descriptions shouid be* included in t/he text ^f the report, 
technical descriptions included in appendices./ The sections should speak 
directly to' the purposes of the evaluation as 'well as being understandable 
to the entire audience. . 

The first section should restate the goals'of the project in as simple 
terms as possible. These terms are expanded upon and elaborated in the* 
specific goal statements which .should follow. Next, a description of the 
particular evaluatif n' procedures used on that grant should be given. Such 
a description should remain nontechnical in the text. A more detailed, - 
elaborate, 'technical description (Including such things as specific computer 
programs emj)loyed, .evidence of all pilot and revised Instruments used, etc..) 
should *be ^eluded in an~ appendix. Following the description of the , 
evaluationl^cheme, a chronicle of the program should be included. All v 
specific n|»es and job descriptions as well as annotated content outlines, 
reference^ etc. may be included in an appendix. Following the program 
descriptidL results from the evaluation procedures should be summarized. 
Finally, ifjcoramendations for program improvements, generalization of training 
efforts atjjjft logical follow-up training programs , should be described . By 
presentini'the material in such a way, the Interested reader may.be able to 
•understand the conceptual framework? of what,. occurred during the course of 
a grant, fcitermlne whether the grain was effective In meeting stated goal*, 



and identify tyr>w t( ? improve the granfe. The more interested or critical 

reader should be able to raal^e \xse of the more technical appendices to 

• ■ * . 

answer any questions remaining following a reading of the text. 

It was the intent of this evaluator to present sprae of the -procedures 

fouftd effective in constructing evaluations for a broad range of grants*. ^ 

It was also my intent to share with- others ray mistakes and ttxxst rations in 

implementing evaluation procedures. I hope that fcfcis document may serve 

as a description to* save someone from encountering similar problems, and 

to'spur others to explore areas and possible solutions £hey otherwise might, 

.... . ._ i ] - . . . — ...... 

have ignored. If any reader should have thoughts to share in this* domain * 
which is continually being ■ explored, I would greatly and sincerely 
appreciate sharing them. 
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Appendix A 



t IntUJ XX NODS AS8ISSMSHT - AXVEMITT OF CONHBCTICOT 
TOt Agency Personnel 



0 

Job Title V 



Agency ■ . - ' K . 

Title XX Agency ] 'Yes flo 

Huaber of years of school coipUttd, ^ 

« 

Nuafcer of clients dealt with In an average day, _ 

All replies will-be treated M confidentiel. 

H? l ^ 0 ^ < T i,l f ? W * hlch «L*«Wfly being considered as topic, for ' 

£Z^ P k I**? 11 ** ••••lone to be conducted In your agency. Could you indicate . 
for aach topic your PRESENT level of knowledge end your DESIRED level of 

vT^?*** 4^,ri b * h#3pful to thlnk about aach topic la to your 

v!lr! P ° W I U1 2 M and *<*> <*•»*>*« Vhan you rata your DESIRED* laval of 
E^iISS!" >MI * lr »J? l »Ul nead to rate eech topic twice, once for PRESENT 
knowledge, onca for DESIRED knowledge. . ' ' 

* \ Little . v ; Much . 

Cer Repair \ . . .Present 1 2 3 . A 5 



Desired 1 2 3*5 

Consuaer Agencies * ... . Present 1 M^-^ * * 3 

Desired 1 2 3. 4 5 

. ^ — 

Consuaer Credit ...... Present 1 , 2 3 



4* • 5 



Desired 1 2 3 ** .4 5 

Door to Door Selling . . .N»£ve*ent 1 2 3 , - 4 t 5 < 

\ Deeired v l . 2 , 3 *4 5 



Agency Personnel 

y • 



-2- 



--VV ^ • 



Food Issues 



Present 
Desired 



ittle - 
Knowledge J 

2 

2 



3 
3 



A* 
4 



Much 

Knowledge 
5 



Food . Steape 



Present 
Desired 



* 2 



■ h ' 



5 
5 



Funerals . 



. . .% . Present 
Desired 



2 
2 



3 - 
3 



S 
5 



Mall Order /' . Present 

Desired 



2 
2 



3 
3 



A 
A 



Purchase of Mtdlcal 
Services 



Present 
Desired 



2 
2 



3 
3 



A 

A 



5 # 

5 



Tenant/Landlord « 



.Present 
Desired 



3 
3 



A 
A 



5 
5 



Werrentles Present 

Desired 



2 



3 
3 



A 
A 



5 
5 



Additional anas for training 



What tine would you prefer to have the training sessions? a. a. i p . aw 

How often would you prefer to have thest training sessions? drily weekly 

n m onthly 

How long do you feelseach session .should be for you to derive the aost benefit? 



2 hours 



A hours 



1 day 



• - A 

What day of the 1*eek do you prefer to have training? 



Agency Personnel 1 -3- , 



Would it be better for you to have the graining in your own agency or at 
a separate aite?. o wn agency .separate site 

Do you heed Job released tine in order to participate in thia type of 
training* _ yes no 

Please give the primary language of your clients: 



JL English, 

% 

X Spanish 



X Other (Specif y j 



100% ^otal 
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Appendix B 



COURSE TITLE 



COURSE NUMBER > 



CREDITS 



^ 



. . i- 

V:* 



DEPARTMENT 



INSTRUCTOR 



YbUR TITliE OR JOB DESCRIPTION 



(1) > WAS THE CONTENT COVERED IN THIS COURSE RELEVANT TO YOUR PRESENT JOS ROLE? 
, a. Always relevant 

. T i , b. Relevant most of tho time ■£ - ■ * 

c. Relevant half of the tine - r ' . * 

, d. Occasionally relevant *r, . " , o * , * * 

* e> Rarely relevant '■ \ 

(2) WAS THE CONTENT COVERED IN THIS COURSE RELEVANT TO FUTURE REQUIREMENTS WHICH 
YOU FORESEE AS BEING INVOLVED IN YOUR JOB ROLE? 

a* Always relevant , , 

b. Relevant mostrof the tine V . 0 

c» Relevant half of the tine 

" <K OccasionaTly/ relevant r 

a. Rarely relevant 



(3) DO YOU 'FEEL THAT THE CONTENT COVERED IN THIS COURSE WILL BE H&PFUL, IN EXPANDING 
YOUR JOB ROLE OR INSTITUTING NEW PROGRAMS OR SERVICES RELATED ^0 VOUR JOB? 

a* ExtremcljTlielpif ul w . \ 

b. Moderately helpful 

c. A bit helpful 



d. Not very helpful 



Not at all helpful 



(4) DO YOU FEEL THAT THE CONTENT COVERED InNmIS COURSE WILL HELP YOU IN DEVELOPING 
YOUR "CAREER? - J ' \ 

a. Extremely helpful , 

b. Moderately helpful , 

4C. A bit helpful * 

d." Not very helpful 

e. Not at all helpful V r 

(3> WAS THE INFORMATION COVERED IN THIS COURSE NEW TO YOU? 

a. Yes, all of the information was new 

" b. Yes f most of the information was new • 

c.^ Some of the information. was new 

d.* No, most -of the information was old * ; » ^ 

e. No, all of the information was old 

(6) DO YOU THINK THAT YOU WOULD HAVE ENCOUNTERED THE INFORMATION COVERED 
IN THIS COURSE IF YOU HAD NOT TAKEN THE COURSE? 

; a. Yes, all of the information 

, b. 'Yes, most. of the information , 

t c. Some of- the,, information 

d. Very ldttl« of the Information ^ • 

a. Probably none of the information h 

(7) WAS ADEQUATE TIME ALLOWED DURING THE" COURSE TO ASK QUESTIONS THAT WERE PERTINENT 
$f> COURSE CONTENT AS IT RELATED TO YOUR JOB REQUIREMENTS? . 

a. ^Yes, alJ /of the, time ) [ ' * 

_ b. Yea * most of the time 

I c. Usually ' * & 

t d. No, hardly ever 
. a. 1 No, none of the time 
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(6) WAS MOST OF THIS COURSE A REPEAT OF THE INFORMATION WHICH YOU HAD ENCOUNTERED 
IN OTHER COURSES? 

. a* Yea IF YES, WHAT COURSE OVERLAPPED WITH THE CONTENT OF THIS OQURSE? 



b. No 



r. 
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(9) WAS THE CLASS TOO LARGE?. 

iii>rt «. Yt« * * - 

; b.' No 

(10) WAS THE COURSE SCHEDULED AT A TIW CONVENIENT TO YOUR NEEDS? 

. a. Yea / ' ;*-;*> V V * ■ ' ■ 

' b. No , 

(11) WAS THE COURSE SCHEDULED ^0 LATE IN THE DAY? 

■ a. Yu . - . n- 

b. No ■ 



(12) WAS THE COURSE SCHEDULED TOQ EARLY IN THE DAY? 

a. - ■ ■ 

b. No V , > 



(13) SHOULD THE COURSE WET FOR A LONGER PERIOD OF TIMS? 

. r *. Hp* '■ 51 .", 

a. Yea 

b. No ' 

(14) SHOULD THE COURSE MEET FOR A SHORTER PERIOD OF TIME* 

a. Yu N 4 

b. No * • 

a ... 

(15) WERE THE COURSE MATERIALS TOO EXPENSIVE-? 

a. Yea IF YE&, GIVE APPROXIMATE COST OF MATERIALS __ JL J 

b. No ^ K 

(16) WERE THE FOLLOWING MATERIALS APPROPRIATE FOR YOUR JOB RELATED NEEDS?, 

* 

BOOKS Yea No _ 

ASSIGNMENTS Yea No 



' AUMO-VISUAL AIDS Yaa -No 

(17) WERE THE FOLLOWING MATERIALS APPROPRIATE FOR THE INFORMATION COVERED 
( IN THE COURSE? 

BOOKS Yaa No 

ASSIGNMENTS * ' * v -Yea " \ ■/ Na 

X A 4UDIO-VISUAL AIDS Yaa No 



(18) IN GENERAL, THIS COURSE WAS: 

a \ ii Excellent 

^ b. Above average 

c. Average 

!_'„'._ > d, Not very good. 

• a* Not worth taking 
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* , * Appendix C 

Summary of Principal Coinponaiits Analysis ot Participant Questionnaire 



factor 1: Materials ' . 82 

,,(16) Were the following materials appropriate for your job 1 ! 

related needs? - - 

BOOKS^ . Yes . No .779 

ASSIGNMENTS : .. Yes ' No , 7tt 

> AUDIOVISUAL AIDS Yes No .450 

(17) Were the following -materials appropriate for the 
Information covered in the course? 

BOOKS f m _ Yes _ . No .772 

ASSIGNMENTS .« Yes No .743 

AUDIOVISUAL AXDS Yes' No ' .473 
* .. 

(15) Were the course materials too expensive? • , t 

J a. YES, GIVE APPROXIMATE COST OF MATERIALS" .464 



< b. NO 

(14) Should the course jneet for a shorter period of tirtie? .330 

a. YES ' ' ■' 4 

b. NO 



1 —- . — - ' 


V 




Items 


\ 

_i * — , — ^ 


Alpha 

* Loading Estimate 
^ c_ 



Edfctor 2 : Course Content 



(2) Was the content covered in this course relevant t< 

future, requirements which you foresee as being involved 
in your job role? - r 



.821 



i a. 



b. 
c. 
d. 
e. 



Always relevant 
Relevarit most of the time 
Releva nt half of the time' 
Occasionally relevant 

9 \ 

Rarely relevant 

m » 



(3)* Do you feel that the content covereS in this course will >750 
be- helpful ia expanding your job role or instituting new / 
programs or services related to your job? 

J 

a. Extremely helpful . - 

b. Mqderately helpful „ 

, c/ A bit helpful t . 

d. Not very helpful 

s Gi Not at all helpful 



(1) Was the content covered in this course relevant to your 
present .job role? 

a. Always relevant 

'* tf. Relevant most of the time 

Relevant t\alf of the tifoe 



.742 



c. 

d. Occasionally relevant 

e. Rarely relevant 



(4) 



Do you feel that the content covered In this course will .692 
help you in developing your career? * 



a. Extremely helpful 

b. Moderately helpful 

c. A bit helpful 

d. Not very helpful 
et- Not at all helpful 



.77 
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(7) Was adequate time allowed during the course to ask questions 
* .-that were pertinent to course content as it related to your . 
^ job requirements? . * <. 

^ a% m Yep, all of the^ime 

b\ Yes, most of the time * 

_ t c: ysuallor" # 

' m d. No, hardly ever 

e. No, none of the tftae 



.355 
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\ . Items 

-c— ,i I • ; : 

Factor 3: Cour$e Scheduling 

(12) Was the bourse scheduled too early iri the day? 
. a. tes 

b.V No • ^ \ 

• . ' . ■ *» * 

(11) Was the course scheduled too late in» the. dayj 

a. Yes 

b. No 3 4 "> " 



(14) Should the course meet for a shorter period of time? 



a. Yes 

b. .No 



(10) Was the' course schedulecUat a time^ convenient to , 
your needs?- t ; '* 

4 a. Yes * 

b. No " J\ 

*(13) Should the course meet for a longer period of time? 

a. Yesi * 

b . No - - 



i 



Items 



Loading 



Alpha 
Estimate 



Factor 4: Use and Uniqueness of the Information Presente'4 

(5) Was the information covered in this course new to you? „847 

a. Yes, all of the information was new * 

* b. Yes, n)08t of the information was* new 

/ * 

c. Some of the information wras new' 



.68 



(6) 



<8> 



(4) 



d. No, most of the information Vas old 

e. No, all of the _ in format io* was old 



Do you think that you would have encountered the 
information covered' in thi«. course if. you had not 
taken th^ course? 



a. Yes, all* of the information 

b. Yes, most of the information 

c. Some of the information' 

d. Very little of the information 



807 




e. t Probably none of the information 



Was most of this course a repeat of the information 
which you had encountered in other courses? 

a. Yes If Yes, what cpurse overlapped with the 

contertt of this course? 

* » 

b • Nt> 



545 



Dp you feef tJfcwtfe-^+re content covered in this course 
will help yoi^in developing your career? 

a^ Extremely helpful * 

b. Moderately helpfyl * ^ 

c. A bit helpful 

d. Not very helpful • # 

^ Not at all helpful , 



.392 



Itedks 



Loading 



* Alpha 
Estimate 



Factor 5: Audiovisual Ai4s and Course Length 

(17) Were the* following materials appropriate for the 
information covered in the course? ' , 



AUDIOVISUAL AIDS 



Ye^s 



No 



(16) Were' the following materials appropriate for your 
job related needs? 



AUDIOVISUAL AIDS 



Yes 



No 



(10) Was the course scheduled at a time convenient 
• to your needs? 



a. Yes 

b. No 



(13) Should the course meet for a longer period of time? 

' m a. -' Yes 

. b. No . ► 



,778 



.764 



.390 



.387 



.56 
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